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ABSTRACT 

Advancements in technology and liberalization of educational opportunities have transformed the knowledge landscape 
into an emergent innovation incubator, where sense -making and creativity in and for ill-structured environments are 
especially relevant. In addition, opportunities are often “found” or “made.” The quality of solutions, however, is mediated 
by different conceptualizations of contexts. As such, we need new teaching-learning models which help students to 
become more effective and creative knowledge builders and creative innovators. In this paper, we present a summary of 
our prior work towards this objective, our framework and the refined assessment rubric derived from our various studies. 
Our design factors/scaffolds are: design thinking as design methodology, metacognitive reflective scaffolds and 
formative assessment as framing technique. We hope our findings can be applied to the learning of Science, Technology, 
Engineering and Mathematics (STEM) and its variants. Future work may involve its application to MOOCs or Cloud 
Computing. 
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1. MOTIVATION 

Crucial to the discovery, creation and sustainability of competitive advantage, knowledge diversity has 
resulted in and encouraged interdisciplinary methodologies. These encompass business, technology and 
human factors. As such, educators are challenged in terms of teaching-learning beliefs and practices with 
regards to how they should best equip/prepare students for the workplace. This is true especially in Multi 
Online Open Coursewares (MOOCs), where students are expected to be independent learners and also where 
dropout rate is sometimes high. 

Rentroia-Bonito, Jorge and Ghaoui (2009) point out that in order to address complex and 
multidisciplinary challenges, development teams need proper design techniques to build effective learning 
experiences. Thus far, they find that the literature does not indicate solid quantitative approaches which 
support learning-centered design where student needs and their immediate and broader contexts are 
considered. Hence, their study explores motivation to e-learn as a variable to designing technology-supported 
learning experiences. They aim to identify what motivation-related variables are critical for student 
engagement in learning online as the basis for a specific, bottom-up and quantitative design technique. Their 
study on the importance of a set of motivation -to-e-learn variables in a real instructional setting has led to an 
exploratory two-factor structure explaining 96% of motivation to e-learn. 

This two- factor structure (presented in Table 1) contributes towards quantitatively understanding and 
cost-effectively improving the link among learning design process, supporting systems and students in terms 
of continuous assessment of what students value most systemically. Courseware accounts for 81% of 
variance whereas organizational communication, accounts for 16% of the common variance. More 
importantly courseware factors explains four times more variance than organizational communication, 
indicating that it has a higher influence on motivation to e-learn and thus deserves more focus and priority 
due to its potential impact. Organizational communication however, influences readiness to e-learn. Among 
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these, the three factors with the highest mean are instructor support, accessibility to contents and 
easy-to-use-interface. We also note that these salient factors include beliefs (the affective domain), indicating 
that any systemic solution has to involve not only courseware and organizational communication, but also 
students’ beliefs/dispositions. This finding is consistent with Perkins and Salomon’s (1992), Bereiter’s 
(1995) and Engle’s (2006) propositions on transfer of learning. 


Table 1. Rentroia-Bonito, Jorge and Ghaoui’s (2009) two-factor structure and factors/variables which load on them. 


Courseware 

Organizational communication. 

Accessibility to contents 

Resource availability 

Security and data protection 

Instructor support 

Personalized feedback 

Institutional support 

Easy-to-use interface 

Feeling part of learning group 

Flexibility in content presentation 

Like studying subject matter 

Aesthetic content presentation 

Belief: E-learning contributes to my learning objectives 

Belief: I can learn this subject online. 

Belief: Adequate communication with instructors 


We are concerned with the design of embedded formative assessment as personalized feedback (part of 
instructor support, accessibility to contents) due to the above scenario and because of our concern with 
cognitive access. Cognitive access extends beyond digital access to include comprehension, and 
interpretation. Interpretation varies depending on the individual's prior learning and culture. As cognitive 
access (sense-making) becomes more critical, we need to provide structure and scaffolds to help students to 
develop their own learning initiative and learning path. We next discuss our suggestions towards this end. 


2. LESSONS LEARNT 

Based on the learning sciences’ literature, we learn that: 

a) transfer of learning is important because if students are able to transfer their learning to different 
contexts, outcomes may be the discovery or creation of opportunities; 

b) promotion of transfer of learning based on modelling (e.g. Kolodner’s (1995) Learning-by-Design, 
Engle’s (2006) socio-cognitive framework) and visualization (e.g. Bhatta, S. and Goel (1997); Ashok’s 
(2011) study on modelling of the acquarium ecosystem) is encouraged. This is because transfer can occur 
at different levels from the elemental to contextual; 

c) goal-seeking analyses and what-if analyses can lead to drill-down analyses. However, it is the 
context, which gives meaning to these analyses. 

To develop design thinking skills and a disposition for social innovations, we have used metacognitive 
reflective scaffolds grounded in inquiry-based formative assessment. 

2.1 Design Thinking as Design Methodology 

Brown and Wyatt (2010) observe that conventional research can point toward incremental improvements. 
However, these do not usually lead to significant breakthroughs. As such, they point out that conventionally, 
designers have emphasized on enhancing the look and functionality of products. However, recently, they 
have begun to utilize design tools in order to address more complex problems. This is because systemic 
problems require systemic solutions. The change in dynamics necessitates broadening approaches in order to 
create entire systems to deliver products and services. For example, social challenges require systemic 
solutions that are grounded not only in the client’s or customer’s needs, but also form and function, as well as 
distribution channels. Without considering the whole system, solutions may not be sustainable. 

As mentioned earlier, design thinking is inherently optimistic, constructive, and experiential. The 
beliefs/epistemology underlying design thinking is illustrated in Figure 1 . 
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Show Don't Tell 

Communicate your vision in an impactful and 
meaningful way by creating experiences, using 
illustrative visuals, and telling good stories. 



Focus on Human Values 

Empathy for the people you are 
designing for and feedback from these 
users is fundamental to good design. 


Craft Clarity 

Produce a coherent vision out of messy 
problems. Frame it in a way to inspire 
others and to fuel ideation. 


Embrace Experimentation 

Prototyping is not simply a way to validate your 
idea; it is an integral part of your innovation 
process. We build to think and learn. 



Be Mindful Op Process 

Know where you are in the design process, 
what methods to use in that stage, and 
what your goals are. 


Bias Toward Action 

Design thinking is a misnomer; it is more about 
doing that thinking. Bias toward doing and 
making over thinking and meeting. 



Radical Collaboration 

Bring together innovators with varied 
backgrounds and viewpoints. Enable 
breakthrough insights and solutions to 
emerge from the diversity. 


D.MINDSETS 


Figure I . Design thinking mindsets 

Source: Stanford’s d. school’s bootcamp bootleg 

Being a Human-Centered Design methodology, design thinking naturally incorporates consumer insights 
as the first design space. In 2001, IDEO transformed from designing consumer products to designing 
consumer experiences. With this change, design thinking develops our ability to a)be intuitive; b)recognize 
patterns; c) construct ideas that have emotional and functional meaning; d) express ourselves in media other 
than words or symbols. In short, IDEO positions design thinking as the integration of feeling, intuition, and 
inspiration with the rational and the analytical. The consequent methodology is thus a system of overlapping 
spaces rather than a sequence of orderly steps. Three main spaces are most well-known: inspiration, ideation, 
and implementation. 

Inspiration emphasizes on empathy through immersive experiences and understanding. These are 
subsequently validated through in-depth and rapid prototyping. Prototyping (transforming ideas into actual 
products and services, which are then tested, iterated, and refined) is crucial as it not only validates, but also 
enables the discovery of unforeseen implementation challenges and unintended consequences; and assesses 
the feedback and viability of new market approaches critical to more sustainable long-term success. 

2.2 Formative Assessment as Framing and Reflective Technique 

Mislevy, Steinberg, Almond and Lukas (2006) point out that effective assessments are key in enabling 
teachers to gather information about students’ knowledge and the depth of their understanding of subject 
matter. In fact, their studies indicate that assessment experts suggest assessments be designed grounded in 
evidentiary reasoning, i.e., the assessment activity is specifically designed to elicit evidence about the student 
knowledge that is of interest. 
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Based on the above literature, in our prior studies (Lee & Wong, 2013, Lee & Wong, 2014a; Lee & 
Wong, 2014b), we embedded metacognitive reflective scaffolds (Schon, 1996) as formative assessments, 
integrated with design thinking within progressive inquiry/problem-based learning. Tangible prototypes are 
produced for the latter two studies consistent with design thinking. Our assessment rubric refined throughout 
our prior work is presented in Table 2 below. 


Table 2. Refined assessment rubric. 



A 

B 

C 

D 

Fun 

Users are addicted/ 

intrinsically 

motivated 

Users find it fun 
and don’t mind 
spending 2-3 hours 
per day at the 
Website/ 
application. 

Users find it 
somewhat fun but 
wouldn’t want to 
go to the Website 
or use the 
application unless 
they have to. 

Users find it 
boring and dread 
to go to the 
Website or use the 
application. 

Interactivity 

Users are actively 
creating and/or co- 
creating objects and 
interactions which 
will result in higher 
quality objects and 
more meaningful 
interaction. 

Users are creating 
and/or co-creating 
objects and 
interactions and 
the outcome are 
more objects and 
more interactions 
which are 
beautiful to them. 

Users are creating 
and/or co-creating 
objects and 
interactions. 

Users are not 
creating and/or co- 
creating objects 
and interactions. 

Sustainability 

Three to four 
dimensions of 
application. 

Two to three 
dimensions of 
application. 

One dimension of 
application. 

Failure to sustain. 

Problem-solving 

Problem and 
hypotheses properly 
formulated across 2- 
3 domains. 

Problem and 

hypotheses 

properly 

formulated across 
2 domains. 

Problem and 

hypotheses 

properly 

formulated across 
1 domains. 

Problem and 
hypotheses 
improperly 
formulated. 

Significance/ 

contribution 

Outcome can 
actually be used in 
society directly. 

Outcome needs to 
be adapted slightly 
before it can be 
used in society. 

Outcome needs 
major revision but 
can be adapted 
before it can be 
used in society. 

Outcome is purely 
adoption of an 
existing solution. 

Novelty 

The outcome is 
original in relation 
to the existing work 
locally and 
internationally. 

The outcome is 
original in relation 
to the existing 
work locally. 

The outcome is 
original in relation 
to the existing 
work in the local 
community. 

The outcome is not 
original. 


The first three criteria are part of what we perceive as contributing to meaningful and sustainable user 
experience. The subsequent three criteria are research-based. Koster’s (2014) theory of fun, analogizes 
people as amazing pattern-matching machines. Once we recognize a pattern, we seek to see it recur. He 
further observes that “Fun is the process of discovering areas in a possibility space” and that “We talk so 
much about emergent gameplay, non-linear storytelling, or about player-entered content. They’re all ways of 
increasing the possibility space, making self-refreshing puzzles”. Moreover, for games to be art, they need to 
be puzzles with more than one right answer, open to the interpretation of the target audience. The challenge 
increases in multiplayer games where there are many instances with variations in player-entered content. 
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Elaboration on these metrics is explained in an extended version of this paper, which illustrates the 
evolution of our studies in the creative industries, contributions and implications, and the refined research 
model. Another paper compares these findings with the findings from the design of assessment for two 
Science-based courses. Software design and testing (Lee, Wong and Lau, 2015) and Robotics (unpublished 
technical report), its research model and the integrated Arts-Science model. 


2.3 Summary of Findings 


From prior studies, we propose an initial conceptual framework to develop generative designs. Our research 
model (Figure 2) considers the following: 

a) learning environments and systems need to focus on theorizing and high-fidelity prototyping; 

b) theorizing can be mediated by framing students’ role as active participants of and contributors to a 
larger intellectual and social community to create intrinsic motivation. This is because the situative 
perspective contextualizes the design of reflective scaffolds to build on prior knowledge triggered by the 
need to design with authentic future applications in mind; 

c) focusing on framing and reframing the context/problem in order to derive more alternatives and 
better solutions. This is based on the grounds that hypotheses are best formulated and tested within 
contexts, driven by the formulation and reformulation of goals/objectives; 

d) to sustain theorizing, learning needs to be fun and interactive; 

e) high-fidelity prototyping is essential during the theorizing, enactment phase; 

f) to be creative, technology needs to allow students to view a problem from different perspectives and 
abstractions to create more novel insights;. 

g) to develop multiple perspectives, we need to extend beyond functional designs to designing user 
experience; 

h) design thinking and metacognitive reflective scaffolds as embedded formative assessments need to 
be integral during lecture and tutorial; 

i) instructors must be brave to allow and look forward to emergent outcomes. 



Figure 2. Research model for this study 
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2.4 Conclusion 

According to Brown (2008), thinking like a designer can transform the way we develop products, services 
and even strategy. In our series of investigations, our objective is to develop a disposition for Social 
Innovations. 

To achieve this, we have aimed to: 

1 ) investigate whether our assessment rubric can be a means to motivate students/designers to learn, to 
model, to analogize, to theorize and to transfer learning through and to different contexts; 

2) develop an ontological affective-socio-cognitive model for context-aware interaction which cares - one 
which will intrinsically motivate and self-direct them to be more open, critical and creative. 

Subsequently, we hypothesize that if we utilize design thinking as our framing methodology, students are 
more likely to emphasize on user experience above functional requirements. However, our sample size is 
small. Hence, findings are not generalizable at that stage. As such, we do not make generalizable claims 
regarding the effectiveness of our scaffolds. At this moment, we can only claim that based on evidential 
findings, our scaffolds (design thinking, metacognitive reflection as embedded assessments and framing), 
help students to be more aware of the context and its relation with their prior knowledge, helps them to think 
broader, make more hypotheses and consider more alternative solutions. 

With this paper, we end our research into the creative industries. We hope to test our initial conceptual 
model in different domains, via different frameworks, interaction platforms and different devices, with 
diverse interdisciplinary collaborators, especially with regards to Science, Technology, Engineering and 
Mathematics and its variants. 

We are keen to collaborate with others within the gamification entrepreneurial framework, continuing 
with our investigations into Web modelling and engineering for the Sciences (Software design and testing) 
and Web modelling and craft for the creative industries while focusing on user experience. UPenn’s 
eCrafting.org and the Mixed Media Lab are referred to as examples. Possible areas of collaboration are: 

a) Enhancing youth's digital lifestyle through social-media-mediated co-creation [Media and 
engagement] 

b) Gamification to improve quality of life to the disabled (visual, learning, auditory). [Assistive tech] 

c) eCrafting for fun, convenience, lifestyle [DigiStyle tech] simulating UPenn’s ecrafting. 

d) Improving comprehension of programming language concepts through the integration of games. 
Lego and programming through visual functional gamified co-design [STEM] 

e) Technology-assisted affective-socio co-design as driver for e-commerce [Media and engagement] 

Ultimately, we hope that our findings in discovering an ontological Affective-Socio-Cognitive Co-Design 

Model and in developing a disposition for Social Innovations can be used to develop a Symbiotic 
Context- Aware Recommender suitable for IoT or Cloud Computing. 
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